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FOREWORD 


This  research  was  conducted  in  support  of  advanced  development  subproject 
ZPN01.06  (Advanced  Navy  Recruiting  System)  and  exploratory  development  task  area 
Z55.521.03G  (Prediction  of  Performance),  it  was  sponsored  by  the  Navy  Recruiting 
Command  and  Headquarters,  Marine  Corps  (MPI-20). 

ihe  report  decribes  the  third  phase  in  the  development  and  evaluation  of  a  test 
battery  for  the  early  identification  of  those  individuals  who  have  the  greatest  likelihood 
of  performing  effectively  as  Navy  and  Marine  Corps  recruiters.  The  first  and  second 
phases  of  the  study  were  described  in  NPRDC  Technical  Reports  76-31  and  79-17.  The 
recruiter  selection  battery  1ms  been  recommended  for  use  by  the  Navy  and  Marine  Corps 
for  predicting  the  success  of  those  individuals  being  considered  for  recruiting  duty  and  is 
currently  being  implemented  by  both  the  Navy  and  the  Marine  Corps. 

Special  gratitude  is  expressed  to  CDR  H,  Levien,  formerly  of  the  U.S.  Navy 
Recruiting  Command,  for  his  excellent  support  throughout  this  study  and  to  all  recruiting 
personnel  who  participated  in  the  research. 

The  contracting  officer's  technical  representative  was  Dr.  Norman  Abrahams. 


JAMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  J.  REGAN 
Technical  Director 


SUMMARY 


Problem  and  Background 

The  U.S.  military  has  a  continuing  need  to  recruit  a  sufficient  number  of  qualified 
young  persons  to  man  the  operating  forces.  To  address  this  problem,  this  Center  is 
conducting  a  three-phase  effort  to  develop  and  evaluate  a  test  battery  for  use  in 
identifying  those  individuals  who  are  most  likely  to  perform  effectively  as  Navy  and 
Marine  Corps  recruiters.  In  the  first  phase,  behaviors  that  contribute  to  effectiveness  as 
a  recruiter  were  identified  and  scales  were  developed  to  measure  tnese  behaviors.  In  the 
second  phase,  a  battery  of  test  instruments  was  validated  on  a  sample  of  267  recruiters, 
using  these  behavior-based  scales.  This  preliminary  validation  revealed  a  number  of 
predictors  that  were  related  to  performance. 

Objective 

The  objective  of  this  research,  the  third  phase  of  this  effort,  was  to  expand  and 
refine  this  test  battery  and  to  determine  its  validity  for  predicting  various  dimensions  of 
recruiter  performance. 

Approach 

The  test  battery  was  revised  by  including  additional  experimental  items  selected  on 
the  basis  of  their  hypothesized  relationship  to  the  underlying  "constructs"  of  the  battery. 
This  revised  battery  was  then  administered  to  194  Navy  recruiters  in  seven  different 
locations.  Two  primary  measures  of  success  were  used:  production  data  compiled  over  a 
6-month  period,  and  ratings  gathered  from  supervisors  and  peers  on  four  aspects  of 
performance.  Analyses  were  conducted  to  evaluate  both  the  precision  with  which  the  new 
items  measured  the  constructs  and  the  extent  to  which  those  items  enhanced  test  battery 
validity. 

Results  and  Conclusions 


Composites  of  new  items  successfully  measured  their  target  constructs,  and,  in  about 
half  the  cases,  enhanced  the  validity  of  these  constructs.  Scales  derived  from  the 
constructs  validly  predicted  the  two  major  indices  of  recruiter  effectiveness:  recruiter 
productivity  (average  number  of  persons  recruited)  and  rated  recruiter  performance. 

Recommendations 


1.  The  Navy  and  Marine  Corps  should  use  the  final  battery  to  aid  in  selecting 
recruiters. 

2.  The  validity  of  the  final  battery  should  be  monitored  to  ensure  its  continued 
usefulness. 

3.  The  performance  rating  materials  should  be  distributed  to  recruiter  supervisors 
and  field  recruiters  to  aid  in  training  and  self-development. 


fnECKDlAG  BaflS  lUiK-WOT  JliJ® 


CONTENTS 


Page 

INTRODUCTION . ). 

Problem  and  Background  .  , .  1 

Purpose . .  .  .  .  . . .  I 

APPROACH . 1 

Factor  Analyses  to  Identify  Valid  Constructs . 1 

Development  of  Revised  Test  Battery  . . 3 

Personality  Items . 3 

Vocational  Interest  Items  . .  3 

Sample . . .  3 

Criteria .  3 

Performance . 3 

Production  . . 4 

Analyses . . 4 

RESULTS . 4 

Criteria .  4 

Predictor  Validation . 6 

Personality  Measures . 6 

Vocational  Interest  Measures .  8 

Personality  and  Interest  Composite  Scales .  9 

Development  . .  9 

Validity- -Rating  Criteria  .  . . 9 

Validity- -Production  Criterion . 10 

DISCUSSION  AND  CONCLUSIONS  . .  13 

RECOMMENDATIONS . 13 

DISTRIBUTION  LIST . 15 


lx 


LIST  OF  TABLES 

Page 

1.  Factors  Identified  for  Each  Performance  Category  .  .  .  .  . .  2 

2„  Navy  Recruiter  Sample . . .  4 

3.  Means,  Standard  Deviations,  and  Interrater  Reliabilities  of 

Peer  and  Supervisor  Ratings  Pooled  Together .  5 

4.  Relationships  Between  Old  and  New  Item  Composites  and  Validities 

of  These  Composites  Against  Recruiter  Performance  Ratings  . .  7 

5.  "Lower  Bound"  Estimates  of  Validity  for  the  Final  Keys  Based 

on  the  Initial  Keys . * .  11 

6.  Validity  of  Final  Keys  for  Predicting  Production . .  13 


x 


INTRODUCTION 


Problem  and  Background 

The  U.S.  military  has  a  continuing  need  to  recruit  a  sufficient  number  of  qualified 
young  persons  to  man  the  operating  forces.  To  address  this  problem,  this  Center  is 
conducting  a  three-phase  effort  to  develop  and  evaluate  a  test  battery  for  use  in 
identifying  those  individuals  who  are  most  likely  to  perform  effectively  as  Navy  and 
Marine  Corps  recruiters. 

In  the  first  phase,  behaviors  that  contribute  to  recruiter  effectiveness  were  identified 
and  scales  developed  to  measure  those  behaviors.1  This  effort  resulted  in  four  scales 
measuring  four  performance  categories:  selling  skills,  human  relations  skills,  organizing 
skills,  and  overall  performance.  In  the  second  phase,  the  scales  were  used  to  validate 
results  of  a  test  battery  consisting  of  personality,  biographical,  and  vocational  interest 
items  that  was  administered  to  a  sample  of  267  recruiters  selected  from  10  Navy 
Recruiting  Districts.2  Scores  obtained  by  sample  members  in  the  four  performance 
categories,  along  with  an  index  'of  recruiter  production  (i.e.,  the  actual  number  of 
personnel  recruited),  provided  the  performance  criteria  for  assessing  the  validity  of  the 
scales  and  their  component  items.  Since  personality  and  vocational  interest  items  were 
analyzed  separately  for  each  of  the  four  performance  categories,  eight  different  item 
pools  resulted.*  These  items  and  categories  served  as  the  basis  for  the  present  effort. 

Purpose 

The  purpose  of  this  research,  the  third  phase  of  this  effort,  was  to  expand  and  refine 
this  test  battery  and  to  determine  its  validity  for  predicting  various  dimensions  of 
recruiter  performance. 


APPROACH 

Factor  Analyses  to  Identify  Valid  Constructs 

Using  the  responses  of  the  267  recruiters  to  the  personality  items,  intercorrelations 
were  computed  within  each  of  the  four  performance  categories  and  the  resulting 
correlation  matrices  were  factor-analyzed  via  the  principal  components  method.  From 
two  to  ten  factors  were  extracted  for  each  category  and  each  solution  was  rotated  to  the 
varimax  criterion.  The  same  procedures  were  employed  with  the  vocational  interest  item 
pools  using  the  Strong-Campbell  Interest  Inventory  (SCII)  (T  325).  Thus,  eight  different 
sets  of  factor  solutions  were  generated,  four  for  the  personality  items  and  four  for  the 
interest  items.  These  factor  solutions  are  provided  in  Table  1. 


‘Borman,  W.  C.,  Hough,  L.  M.,  &  Dunnette,  M.  D.  Development  of  behaviorally- 
based  rating  scales  for  evaluating  the  performance  of  U.S.  Navy  recruiters  (NPRDC  TR 
76-31).  San  Diego:  Navy  Personnel  Research  and  Development  Center,  February  1976. 
(AD-A022  371)  K 

2Borman,  W.  C.,  Toquam,  3.  L.,  &  Rosse,  R.  L.  An  inventory  battery  to  predict  Navy 
and  Marine  Corps  recruiter  performance:  Development  and  validation  (NPRDC  TR  79- 
17).  San  Diego:  Navy  Personnel  Research  and  Development  Center,  May  1979.  (AD-A069- 
371) 

*Items  were  not  constrained  to  appear  in  only  a  single  pool.  In  fact,  some  items 
appeared  in  all  four  interest  or  personality  item  pools. 
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Table  1 


Factors  Identified  for  Each  Performance  Category 


Performance 

Categories 

Factor/ Construct 

Personality  Items 

Selling  Skills 


1.  Good  impression. 

2.  Impulsive,  carefree  vs.  order,  planning  ahead,  system¬ 
atic,  level-headed. 

3.  Enjoying  being  center  of  attention,  leading,  showing 
off,  and  speaking  before  a  group. 

4.  Working  hard  and  with  confidence,  being  happy  vs. 
being  unhappy,  giving  up  easily,  disgruntled  about  life. 


Human  Relations 
Skills 


Organizing  Skills 


1.  Preference  for  working  with  and  being  with  people. 

2.  Spontaneity,  impulsivity,  "fast  and  careless,"  rebel¬ 
lious,  tendency  to  have  bad  moods. 

3.  Unhappy,  lack  of  confidence,  disgruntled  about  life. 

4.  Ambitious,  working  hard,  pushing  self. 

1.  Order,  planning  ahead,  well  organized  vs  impulsive, 
acting  without  thinking,  "fast  and  careless." 

2.  Leading  and  influencing  others,  giving  orders,  demand¬ 
ing  of  self,  ambitious,  dominant. 

3.  Unhappy,  discouraged,  doing  little  in  life,  giving  up 
hope,  feeling  useless. 

4.  "Bad  actor,"  was  unruly  and  rebellious  in  school, 
unsocialized.3 


Overall 

Performance 


1.  Doing  more  than  expected  vs.  giving  up,  working  just 
hard  enough. 

2.  Impulsive,  "fast  and  careless"  vs.  order,  methodo¬ 
logical,  planning  ahead. 

3.  Leading  and  influencing  others,  dominant,  strong  per¬ 
sonality. 

4.  Good  impression  vs.  admitting  occasional  meanness, 
grouchiness,  disgust  with  self,  discouragement,  use¬ 
lessness,  bad  mood. 

5.  People  oriented,  liking  to  be  around  others  and  close 
to  others,  open  to  other  people. 


Voca  tional  Interest  Items 


Selling  Skills 


1.  Interest  in  extroverted,  dominant,  leadership  activities 
and  occupations. 

2.  Interest  in  occupations  involving  attention  to  detail.3 

3.  Intsrsst  in  law  and  politics. 

4.  Interest  in  sports  and  competitive  activities. 


Human  1. 

Relations  2. 

Skills  3. 

4. 

5. 

6. 

Organizing  I. 

Skills  2. 

3. 

4. 

Overall  I. 


Performance 

2. 


3. 

4. 


J. 


Interest  in  dominant,  extroverted,  social  activities. 
Interest  in  teaching  and  counseling. 

Interest  in  "feminine"  occupations  and  activities. 
Interest  in  newspaper  reporting  and  foreign  service. 
Interest  in  sports  and  competitive  activities. 

Interest  in  religion  and  in  being  around  the  sickly. 

Interest  in  politics  and  high  level  management  jobs. 
Interest  in  bookkeeping,  statistical,  and  detail  work. 
Interest  in  "feminine"  occupations  and  activities. 
Interest  in  leadership  and  responsibility. 

Interest  in  law  and  poltics,  and  management  occupa¬ 
tions  and  activities. 

Interest  in  activities  and  occupations  that  require 
extroversion,  dominance,  responsibility,  and  leader¬ 
ship. 

Interest  in  sports  and  competitive  activities. 

Interest  in  teaching  and  counseling. 

Interest  In  "feminine"  occupations.3 


aThese  constructs  related  negatively  to  their  target  performance  criteria. 
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Development  of  Revised  Test  Battery 


Personality  Items 

Existing  personality  items  were  reviewed,  and  those  items  that  correlated  .10  or 
greater  with  the  target  performance  category  were  included  in  a  new  trial  battery.  This 
resulted  in  the  selection  of  315  items- -55,  85,  95,  and  80  being  selected  for  the  selling 
skills,  human  relations  skills,  organizing  skills,  and  overall  performance  categories 
respectively. 

To  generate  new  personality  items,  several  personality  inventories  were  reviewed  for 
scales  conceptually  related  to  one  or  more  of  the  17  constructs  identified  in  the  factor 
analyses  (see  Table  1).  Each  item  was  then  examined  and  those  that  appeared  to  tap  these 
same  constructs  were  selected.  For  example,  for  the  construct,  "Leading  and  influencing 
others  ..."  (No.  3  under  overall  performance),  the  item  "I  try  to  control  others  rather 
than  permit  them  to  control  me"  from  the  Dominance  Scale  of  the  Personality  Research 
Form*  was  selected.  In  all,  83  items  were  selected.  In  addition,  26  new  personality  items 
of  the  same  general  type  as  the  above  example  were  written,  each  targeted  toward  one  of 
the  identified  constructs.  Thus,  the  revised  test  battery  included  424  personality  items. 

Vocational  Interest  Items 


Existing  vocational  interest  items  were  selected  for  the  trial  battery  based  upon  one 
of  the  following  criteria:  (1)  r  >  |.12|  in  the  sample  of  267  recruiters  and  r  >  .00  in  the 
same  direction  in  a  pilot  test  sample  of  62  recruiters  in  the  same  study,  or  (2)  r  >  |  .24  j  in 
the  sample  of  62  and  r  >  ±  .08  in  the  same  direction  in  the  sample  of  267.  This  screening 
procedure  resulted  in  the  selection  of  202  items- -39,  62,  48,  and  53  being  selected  for  the 
selling  skills,  human  relations  skills,  organizing  skills,  and  overall  performance  categories 
respectively. 

In  addition,  49  new  vocational  interest  items  were  written  to  tap  the  19  constructs 
derived  from  the  factor  analyses  (see  Table  1).  For  example,  the  item,  "College  football 
coach"  (to  which  the  respondent  answers  "like,"  "indifferent,"  or  "dislike")  was  written  to 
measure  the  constructs  related  to  sports  interests,  and  the  item,  "Keeping  track  of 
statistics  for  baseball,  football,  etc.,"  was  written  to  tap  the  construct  dealing  with 
interest  in  detail  work.  Thus,  the  revised  test  battery  included  251  vocational  interest 
items. 

Sample 

The  revised  test  battery  was  administered  to  a  sample  of  194  Navy  recruiters.  The 
distribution  of  recruiters  by  geographic  location  is  provided  in  Table  2. 

Criteria 


Performance 


The  rating  scales  developed  by  Borman  et  al.  (1976)  were  used  to  evaluate  the 
performance  of  the  recruiters  in  this  sample.  Ratings  were  obtained  from  supervisors  and 
peers. 


''Jackson,  D.  N.  Personality  Research  Form.  Goshen,  NY:  Research  Psychologist 
Press,  Inc.,  1965. 
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Table  2 


Navy  Recruiter  Sample 


District 

N 

Albany 

30 

Baltimore 

28 

Boston 

30 

Dallas 

27 

Detroit 

28 

Minneapolis 

22 

San  Antonio 

29 

Total 

194 

Production 


Previous  attempts  to  develop  scales  to  predict  production,  as  reflected  by  the  actual 
number  of  accessions,  have  been  largely  unsuccessful.  However,  since  production  is 
often  perceived  as  an  important  measure  of  recruiter  effectiveness,  records  were 
gathered  for  recruiters  in  this  sample.  Production  was  monitored  over  a  6-month  period 
to  determine  the  average  number  of  accessions  per  month. 

Analyses 


1.  Performance  ratings  were  factor  analyzed  to  determine  whether  they  related  to 
the  four  performance  categories.  Factor  scores  were  computed  for  each  recruiter  in  the 
sample. 


2.  Correlational  analyses  were  conducted  (1)  to  assess  how  precisely  the  new  items 
measured  their  target  constructs  and  therefore  how  clearly  we  understood  these  con¬ 
structs,  and  (2)  to  evaluate  the  validity  of  the  constructs  as  indicators  of  Na'^y  recruiter 
effectiveness. 


RESULTS 


Criteria 


Table  3  presents  the  means  and  standard  deviations  of  performance  ratings,  along 
with  the  peer/supervisor  interrater  reliabilities.  Since  the  mean  ratings  range  from  6.47 
to  7.48  (on  a  1-10  scale),  it  appears  that  leniency  error  is  not  a  serious  problem  here. 
Also,  the  standard  deviations  indicate  that  the  range  of  the  ratings  is  not  severely 
restricted .  Finally,  the  interrater  reliabilities  appear  to  be  acceptable  and  comparable 
to  thor«  obtained  in  the  previous  recruiter  study  (Borman  et  al.,  1979). 

Table  3  also  includes  the  results  of  the  factor  analysis  performed  on  these  ratings. 
The  fact  that  the  three-factor  solution- -selling  skills,  human  relations  skills,  and 
organizing  skills--is  similar  to  solutions  generated  previously  (Borman  et  al.,  1979)  lends 
additional  stability  and  meaningfulness  to  this  dimensional  structure  for  describing  Navy 
recruiter  performance.  The  three  factors  were  defined  as  follows. 
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Reliability  coefficients  are  intraclass  correlations  reflecting  the  reliability  of  the  mean 
racings  pooled  across  the  two  sources. 


1.  Selling  Skills— Selling  Navy  effectively  to  prospects;  displaying  confidence  and 
effectiveness  in  the  recruiting  sequence--prospecting,  selling,  and  closing. 

2*  Human  Relations  Skills — Establishing  and  maintaining  good  interpersonal  rela¬ 
tions  with  prospects,  recruits,  and  persons  in  the  community. 

3.  Organizing  Skills— Planning  ahead  and  organizing  time  efficiently;  completing 
paper  work  accurately  and  on  time. 

The  interrater  reliabilities  of  the  factor  scores  completed  were  .62,  .48,  and  .65, 
respectively,  sufficiently  high  to  allow  the  factor  scores  to  represent  individual 
recruiters'  effectiveness  in  three  different  aspects  of  Navy  recruiting.  Therefore,  the 
three  factor  scores  and  the  highly  reliable  overall  performance  rating,  which  provided  a 
summary  effectiveness  measure,  were  used  as  criteria  to  evaluate  relationships  between 
the  various  personality/vocational  interest  constructs  and  recruiter  performance. 

Predictor  Validation 


Personality  Measures 

Table  4  provides  the  validities  of  composites,  consisting  of  old  items,  new  items,  and 
old-plus-new  items,5  measuring  the  constructs  identified  in  the  factor  analysis  (see  Table 
1),  as  well  as  the  convergent  validity  indices  (i.e.,  correlations  between  the  old  and  new- 
item  composites  targeted  to  measure  the  same  constructs).  Since  the  median  correlation 
between  old  and  new  items  measuring  each  construct  is  .56  (p  <  .001),  indicating 
reasonably  high  convergent  validity,  it  appears  that  the  new  items  are  tapping 
substantially  the  same  constructs  as  the  old  ones. 

Further  discriminant  validity  of  the  new-item  composites  is  dearly  demonstrated: 
Convergent  validity  correlations  are  greater  than  correlations  between  each  new-item 
composite  and  old-item  composite  measuring  different  constructs  for  all  but  two  of  56 
such  comparisons.6  These  results  suggest  that,  in  most  cases,  the  attempts  to  understand 
and  supplement  the  original  item  constructs  were  successful. 

The  validities  for  the  old-item  composites  in  the  previous  sample  should,  of  course, 
be  high  because  the  items  contained  in  those  composites  were  selected,  in  part,  according 
to  their  validities  in  that  sample.  Validity  coefficients  of  old-item  composites  in  the 


sEach  "old-item"  composite  corresponding  to  a  factor  (construct)  was  formed  by  unit¬ 
weighting  responses  to  all  items  loading  suffidently  highly  on  that  factor  (and  not  highly 
on  any  other  factor)  in  the  N  =  267  sample.  In  other  words,  an  old-item  composite  for  a 
factor  consisted  of  the  unit-weighted  marker  items  for  that  factor.  Each  new-item 
composite  (one  for  each  construct)  was  developed  by  simply  unit- weighting  responses  to 
the  new-item  targeted  toward  that  construct. 

*To  help  explain  the  discriminant  validity  analysis:  For  example,  the  .32  convergent 
validity  coefficient  for  the  "Good  impression"  construct  is  greater  than  the  correlations 
between  the  new-item  composite  for  "Good  impression"  and  the  old-item  composites 
representing  (1)  "Impulsiveness,"  (2)  "Leading  and  showing  off,"  and  (3)  "Working  hard," 
and  this  same  pattern  of  discriminant  validity  obtains  for  almost  all  of  the  other  new-item 
composites. 
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Holj tionships  ftc  tween  Old  arxf  New  Item  Composites  and 
Validities  of  1  hese  Cixn posit es  wgainst  Ke  roster  Pc  fonnarve  Ratings 


Convergent 
Validities.: 
Correlation 
I3ctwecn  Olu 
and  hew  Items 

Old  1  tern 
Con.posi  t^s 
in  Previous 

Validities  Against 

Old  Item 
Composites 
in  Present 

Performance  Ratings 

New  Item 

Composites  Old  Plus  New 

m  Present  Item  Composi tes 

Category 

Cons  true  ta 

v^ld  Items 
IN) 

New  Iter 
IN) 

ns  Composites 

Sample 

Sample 

Samp!" 

in  Present  Sample 

— 

Personality  Items 

filing 

Good  impressio.1 

4 

.12*'' 

.  IS* 

.20*  * 

.05 

.  IS* 

Skills 

‘mpulsivt 

Enjoying  being  center 

15 

32 

.72*  * 

.23** 

.01 

.03 

.0) 

of  attention 

IS 

»» 

.59** 

.29*  • 

.26** 

.22** 

.27** 

Working  hard 

15 

21 

.  io** 

.17* 

.03 

.  1  1 

Human 

People  oriented 

n 

34 

.  56*  J 

,23*> 

.12 

.08 

.11 

Relations 

Skdls 

Spontaneity,  imp*  Isivity 
Unhappy,  Jack  of 

7 

<0 

.  |l»* 

.17*  * 

.08 

.77" 

.22** 

'•not  idenc  e 

Ambitious,  working 

7 

6 

.i*0*  * 

-.IS* 

-.14° 

-.  16* 

.17* 

hat  d 

10 

1  5 

.46** 

.  30  •  * 

.24-“ 

.  *  •*  * 

.29** 

Organizing 

Skills 

•Vder,  planning  ahead 
Leading  and  influencing 

25 

v 

.72" 

.  If  •* 

. » 2 

.22*  * 

.  17* 

others  . 

lb 

*2 

.6.4*  * 

,7b" 

.  10 

.12 

.  12 

Unhappy,  discouraged 
Had  actor’* D 

1  3 

10 

.  30** 

-,2S" 

.01 

.04 

.00 

6 

10 

.21** 

-.14* 

-.05 

-.04 

.06 

Overall 

Working  lyjrd 

8 

12 

.  5J»* 

.28"* 

.03 

.  16* 

.  1  3 

Performance 

Impulsive 

Leading  and  influencing 

10 

27 

.61  ** 

.  1 

.05 

.07 

.08 

others 

1 3 

37 

.64*  * 

.28** 

.26** 

.31** 

.  n** 

Good  impression 

9 

i  1 

.<>?** 

- 25 *  • 

.12 

-.02 

.08 

People  oriented 

\  l 

35 

.  57*  * 

,7b" 

.07 

-.04 

.00 

All  Pcffo.-- 

Selling  skills 

S3 

*8 

.7)"* 

.  47  *  • 

.  ?5#  * 

.19** 

.23** 

mancc  Cate- 

Human  relations  skills 

55 

85 

.57** 

.44*  • 

.18** 

.25** 

.  ?4  *  * 

gory  Con- 

Organitng  skills 

95 

89 

.  f*9  « 

.45*  * 

.09 

.22" 

.  1  5° 

struct* 

Overall  performance 

80 

122 

.7  ■ 

.46*  * 

.21-* 

.2 1*- 

.22** 

Vocational  Interest  Items 


Soiling 

Extroverted  intereits  b 

9 

7 

.69*° 

.23*  * 

.20*  • 

.20** 

.22** 

Skills 

Interests  in  detail  work 

5 

2 

.65** 

- .  18® 

.00 

-.03 

.01 

Law  and  political  interests 

1  i 

2 

.75** 

.21  •• 

.13 

.  14* 

Sports  Interests 

6 

6 

.51  ** 

.2)*’ 

-.0) 

.03 

.00 

Human 

Extroverted  interests 

16 

4 

.56** 

.2  3®  * 

.23** 

.19** 

.24** 

Relations 

Interest  jn  lea<  hmg 

4 

1 

. 78*  * 

.17* 

.13 

.07 

.  1 1 

Skill! 

"Feminine11  interests 

.8 

5 

.  53*  • 

.24*  * 

.05 

.07 

.07 

Interest  in  newspaper  wotU 

6 

2 

.  )«** 

.!S* 

.  16* 

.06- 

.18* 

Sports  interests 

3 

6 

.62*  * 

.17* 

.20** 

.20*  * 

.22** 

Religious  interests 

5 

4 

.71  •’ 

.22" 

.09 

.13 

.12 

Organizing 

Interest  m  politics 

16 

4 

.76** 

.27" 

.  16* 

.It 

•  1  5  • 

Skill! 

Interest  in  detail  work 

10 

3 

.24*  * 

.01 

-.01 

.02 

'’Feminine'1  interests 

4 

2 

,52*« 

-.19*  • 

.08 

-.02 

.05 

Leadership  interests 

y 

4 

.  58*  • 

.  29*  * 

*0! 

•  .04 

-.01 

Overall 

Law  and  political  interest 

1  1 

4 

.71  *• 

.•»»*»« 

.  i0 

71*  » 

.  14* 

Performance 

Extroverted  interests 

14 

7 

.69*  • 

.2 

.23** 

.22" 

.24** 

Sports  m'erevs 

6 

6 

.72** 

.21*  ■ 

■  H 

.03 

.  10 

Interest  in  teaching  ^ 

5 

2 

.73’* 

.  i  9®  • 

.  I ! 

.00 

■  OS 

"Feminine*1  interests 

3 

0 

.14* 

-.03 

,03 

AH  Pet  for- 

Selling  skills 

19 

[7 

.62** 

.35** 

.  19*  • 

.20*  * 

.21** 

malice  Cate- 

Human  relations  skills 

62 

22 

69** 

.320" 

.20** 

.21“* 

.22" 

gory  Con- 

Organizing  skills 

4S 

1  l 

.  6  3  *  * 

.  37*  * 

1  4  • 

.08 

.17 

struct* 

Overall  performance 

5  3 

19 

.73*  * 

.  3*» 

.21** 

.  »9** 

.22** 

Together 

aSee  Tabto  I  ?o<-  more  mmpicte  definitions  of  these  constructs. 

^Tbes*  constructs  should  relate  negatively  to  tltetr  target  performance  criteria. 

cFor  each  performance  tategory,  a  single  composite  was  twined  b»'  unit  weighting  items  contained  in  the  4,  \  or  6  composites  targeted  toward  that 
criterion. 

•p  <  .03. 

**p  •  .01. 
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present  sample  {N  =  194),  however,  provide  unbiased  cross-validity  estimates  for  those 
composites.  Wniie  many  of  these  cross- validities  are  much  reduced  in  magnitude,  several 
suggest  reasonably  high  a: id  consistent  relationships  between  the  constructs  and  perfor¬ 
mance.  These  relationships  will  be  elaborated  upon  more  fully  below. 

Regarding  the  validity  of  the  new  personality  items,  the  data  in  Table  4  indicate  that 
all  but  2  of  the  17  validities  for  the  new-item  composites  are  in  the  proper  direction.  In 
audition,  7  of  these  17  validity  coefficients  are  significantly  different  from  zero  in  the 
predicted  direction  at  the  .05  level  or  greater,  indicating  at  least  modest  validity  against 
their  target  performance  criteria.  For  9  of  the  17  constructs,  the  new- item  composites 
provide  validities  higher  than  those  provided  by  the  oid-item  composites.  Also,  for  9  of 
the  17  con.  "ructs,  composites  consisting  of  the  old  plus  the  new  items  show  higher 
validities  than  do  the  composites  cc  taining  the  old  items  alone.  In  other  words,  the 
inclusion  of  the  new  items  in  composites  enhanced  the  validity  of  those  composites  for 
slightly  more  then  half  th  2  constructs. 

Table  4  also  reveals  the  pp~sor«iiity  constructs  most  highly  correlated  with  various 
aspects  of  recruiter  effectiveness.  "Making  a  g'-  d  impression'1  and  "Enjoying  being  the 
center  of  attention"  are  the  constnxrts  thet  correlated  highest  with  selling  skills.  For  the 
human  relations  ski'ls  category,  "Spontaneity,  impulsivity"  and  "Ambitious,  working  hard" 
had  the  highest  positive  correlations;  in  addition,  "Unhappy,  lack  of  confidence" 
consistently  related  negatively  to  human  relation,,  effectiveness.  The  only  construct  that 
related  well  to  organizing  skills  was  "Order-  planning  ahead."  Finally,  for  the  overall 
performance  category,  the  most  highly  correlated  construct  was  "Leading  and  influencing 
others." 

Vocational  Interest  Measures 

The  results  of  the  analyses  for  vocational- interest  composites  are  also  presented  in 
Table  4.  Again,  the  new  items  witten  to  measure  the  target  constructs  do,  in  fact, 
appear  to  be  successfully  measuring  those  constructs.  The  median  correlations  between 
the  new-item  composites  and  the  oid-item  composites  (*he  convergent  validities)  is  .Sir  (p 
<  .001).  Also,  discriminant  validity  is  excellent,  in  all  con  ,  risons  between  convergent 
validities  and  correlations  between  new-and  old-item  composites  not  intended  to  measure 
the  same  construct,  the  magnitude  of  the  convergent  vahdit:e„  is  greater.  Thus,  as  was 
the  case  in  the  personality  domain,  the  levels  of  convergent  and  discriminant  validity 
noted  here  indicate  accurate  conceptualization  and  rei  itively  precise  measurement  of 
several  vocational-interest  constructs. 

Finally,  the  pattern  of  validities  appearing  for  the  interest  hems  is  very  similar  to 
the  pattern  noted  with  the  personality  items.  Old-item  composites  relate  well  to  perfoi- 
mance  in  the  original  (N  =  267)  sample,  and,  in  general,  these  relationships  are  .ower  in 
the  present  (N  =  194)  sample.  About  half  of  the  validities  for  the  compr  rates  containing 
new  items  are  higher  than  the  validity  coefficients  provided  by  old-item  composites  (7  of 
16,  2  ties).  Further,  for  11  of  the  18  constructs7  considered  here,  validities  of  composites 
consisting  of  old  and  new  items  polled  together  are  higher  (in  the  intended  direction)  than 
the  validity  coefficients  obtained  when  the  old  items  alone  form  the  composites. 


’Although  Table  l  shows  19  constructs,  one  (under  overall  performance)  was  excluded 
because  no  new  items  were  developed  for  that  construct. 
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The  constructs  reflecting  relatively  high  and  consistent  relationships  with  perfor¬ 
mance  criteria  are  (i)  interests  in  extroverted,  dominant,  social,  and  leadership  activities 
and  occupations,  (2)  interests  in  sports  and  competitive  activities  (for  the  human  relations 
category  only),  and  (3)  to  a  somewhat  lesser  extent,  interests  in  law  and  political 
activities. 

Personality  and  Interest  Composite  Scales 

Development 

Validities  for  the  original  personality  and  vocational  interest  items  were  assessed, 
and  items  were  included  in  the  final  scoring  keys  if  they  demonstrated  consistent  validity 
across  the  two  concurrent  validity  samples  (N  =  267  and  N  =  194).  New  items  were  also 
considered  for  the  scoring  keys  if  they  demonstrated  good  validity  in  the  present  sample, 
provided  the  direction  of  the  validities  was  consistent  with  prediction  made  when  these 
items  were  selected  or  written.  For  example,  the  validity  of  the  personality  item,  "1  work 
hard  even  if  1  don't  think  it  will  get  me  ahead,"  was  written  to  measure  the  construct, 
"Working  hard."  The  item's  validity  proved  to  be  -.19  in  the  hypothesized  direction  and 
large  enough  in  magnitude  for  the  item  to  be  considered  for  inclusion  in  the  final  keys. 

To  assess  potential  response  distortion,  all  the  original  items  demonstrating  validity 
were  administered  to  a  sample  of  recruiter  applicants  (N  =  131).®  For  some  items,  the 
responses  of  the  applicant  group  were  in  a  much  more  socially  desirable  direction  than 
were  those  of  the  validation  sample.  These  items  were  eliminated  from  consideration  for 
the  final  keys  because  they  appeared  to  be  easily  faked  and  therefore  very  possibly 
undesirable  for  use  in  an  actual  selection  setting. 

Unfortunately,  similar  response  rate  data  were  not  available  for  the  new  items 
because  the  battery  administered  to  the  applicant  sample  did  not  contain  these  items. 
Thus,  new  items  that  had  sufficiently  high  validity  to  be  considered  for  the  keys  were 
rejected  if  they  appeared  to  be  similar  in  content  to  those  old  items  tliat  had  been 
rejected  on  the  basis  of  differences  in  response  rates.  The  resulting  scoring  keys  fcr  the 
personality  inventory  contain  110  items  and  the  vocational  interest  keys,  60  items. 

In  aii,  the  keys  yield  four  separate  scores  that  can  be  computed  from  responses  to  the 
final  battery,  one  for  each  effectiveness  category  (selling  skills,  human  relations  skills, 
organizing  skills,  and  overall  performance). 

Validity- -Rating  Criteria 

To  provide  additional  perspective  on  the  level  of  validity  associated  with  the  final 
keys,  scores  on  each  key  were  correlated  with  performance  in  the  appropriate  category  in 
the  present  sample  (N  =  194).  These  correlations  are  .43,  .46,  .40,  and  .43  respectively  for 
the  four  performance  categories,  indicating  impressive  relatioaships  between  performance 
and  inventory  responses  to  the  keyed  items.  These  correlations  may  capitalize  somewhat 


aThe  applicant  sample  consisted  of  131  fleet  personnel  who  had  volunteered  for 
recruiting  duty.  Members  of  the  sample  completed  the  same  inventory  battery  that  was 
administered  to  the  previous  concurrent  validity  sample  (N  =  267).  The  administration  set 
was  "for  real,"  with  instructions  indicating  that  scores  obtained  on  the  inventory  might  be 
used  to  select  or  reject  testees.  In  fact,  however,  the  battery  was  not  used  as  a  selection 
device. 
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on  chance  relationships  between  scores  on  the  keys  and  performance  because  items  were 
selected  for  the  final  keys  on  the  basis  of  their  correlations  with  performance  in  this 
sample  and  in  the  previous  sample  (N  =  267).  However,  relatively  little  ’'shrinkage"  in 
these  validities  is  anticipated  because  only  items  with  consistent  validity  across  the  two 
samples  were  selected.  Accordingly,  these  correlations  may  be  considered  "upper  bound" 
estimates  of  validity,  although  it  is  likely  that  they  are  not  gross  overestimates  of  the 
final  keys'  validities. 

"Lower  bound"  estimates  were  also  computed.  Items  that  proved  to  be  valid  in  the 
previous  sample  (the  "old  items")  were,  for  each  inventory  (i.e.,  personality,  vocational 
interest,  and  sales  effectiveness  inventories),  assembled  into  unit-weighted  scales  and  the 
validity  of  each  scale  was  computed  in  the  present  sample.  Then,  disregarding  inventory, 
a  single  composite  of  these  scales  was  formed  for  each  performance  category  and  the 
validity  of  each  of  these  composites  was  estimated  using  the  following  formula;9 

r  total  =_JVcrt _ 

y  s*y  i  *  2ErijwiwjCTi°j 


where: 


w*s,  the  weights,  were  all  set  to  1.0 
a  =  standard  deviations  of  each  composite 
r^.  =  validity  of  each  composite 
r..  =  correlations  between  composites 


These  composite  validities,  shown  in  Table  5,  are  considered  lower-bound  estimates 
because  they  include  many  items  that  were  later  rejected  from  the  final  keys  due  to 
insufficient  levels  of  validity.  Further,  they  do  not  contain  the  new  items  that  proved  to 
be  valid  in  the  present  sample.  Overall,  the  validity  coefficients  obtained  using  upper- 
bound  estimates  of  the  final  keys  and  the  lower-bound  estimates  of  the  initial  keys 
indicate  acceptable  levels  of  validity. 

Next,  the  practical  impact  of  using  these  keys  for  selection  was  examined  to 
determine  the  relationship  between  inventory  scores  and  the  likelihood  of  being  an  above- 
average  performer  on  each  criterion.  This  examination  assumed  actual  validities  midway 
between  the  upper-  and  lower-bound  estimates.  Results  are  provided  in  Figure  1,  which 
shows,  for  example,  that  72  percent  of  the  recruiters  in  the  upper  20  percent  on  the 
Selling  Skills  predictor  were  successful,  compared  to  only  29  percent  of  those  in  the 
lowest  20  percent. 

Validity- -Production  Criterion 

In  Borman  et  al.  (1979),  it  was  argued  that  production  "numbers"  do  not  necessarily 
reflect  a  recruiter's  true  contribution  to  the  recruitment  effort,  especially  since  adminis¬ 
trative  reporting  of  accessions  may  vary  liom  L  -ation  to  location,  thus  contaminating 
production  statistics.  Production,  however,  is  ofter  used  as  ti.e  sole  index  of  recruiter 
effectiveness.  Therefore,  scores  on  each  of  the  four  final  keys  (in  the  present  sample) 


sGuilford,  3.  P.  Fundamental  statistics  in  psychology  and  education  (4th  Ed.).  New 
York:  McGraw-Hill,  1965. 
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Figure  1.  Percent  successful  recruiters  expected  where  upper  50  percent  are  con¬ 
sidered  successful. 


were  correlated  with  each  recruiter's  raw  production  data  (mean  number  of  accessions 
during  the  6-month  period  of  October  1977  to  March  197S).  As  shown  in  Table  6,  the 
correlations  for  the  four  categories  are  .22,  .23*  .13,  and  .26  respectively,  with  three  of 
the  four  being  statistically  significant  (p  =  .01).  This  suggests  that  the  final  keys  will 
successfully  predict  production  in  Navy  recruiting.  The  magnitude  of  these  correlations  is 
actually  Impressive,  given  the  amount  of  error  likely  to  be  present  in  the  production  data. 
Thus,  the  final  keys  yield  four  separate  scores  that  may  be  used  to  forecast  production  as 
well  as  to  predict  performance,  the  criteria  for  which  they  were  specifically  devc  oped. 

When  the  four  separate  scores  are  summed  into  s  composite,  they  correlate  .27  with 
production.  The  practical  significance  of  this  relationship  is  also  depicted  in  Figure  1.  As 
shown,  66  percent  of  the  recruiters  scoring  in  the  top  20  percent  were  in  the  upper  50 
percent  in  production,  compared  to  39  percent  of  those  scoring  in  the  lowest  20  percent. 
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Table  6 


Validity  of  Final  Keys  for 
Predicting  Production 
(N  -  194) 


Predictor  K  ey 

Correlation 

with 

Production 

Selling  Skills 

.22* 

Human  Relations  Skills 

.23* 

Organizing  Skill 

.13 

Overall  Performance 

.26* 

*p  <  .01. 


To  examine  the  utility  of  these  selection  procedures  in  a  slightly  different  way, 
recent  data  indicated  that  the  average  monthly  production  per  recruiter  is  2.5  accessions 
with  a  standard  deviation  of  1.3,  The  estimated  monthly  production  for  those  recruiters 
who  score  in  the  top  50  percent  on  the  predictor  composite  is  2.80;  accordingly,  this 
group's  average  production  would  be  fully  12  percent  greater  than  the  current  overall 
average. 


DKCUSSIOM  AMD  CONCLUSION© 

The  primary  purpose  of  this  research  has  been  to  develop  measures  predictive  of 
Navy  and  Marine  Corps  recruiter  performance.  The  procedures  presented  here  succeeded 
in  identifying  personality  and  vocational  interest  constructs  related  to  one  or  more 
aspects  of  recruiter  effectiveness,  and  attempts  to  develop  additional  parallel  measures 
of  these  constructs  were  successful. 

The  process  used  in  this  study  of  isolating  individual  differences  constructs  that  are 
important  for  job  performance  has  provided  one  way  to  gain  understanding  of  individual 
differences/job  performance  linkages.  These  techniques  have  aided  in  the  development  of 
a  test  battery  for  predicting  recruiter  effectiveness  and  should  be  considered  for  use  in 
other  research  and  development  programs. 


RECOMMENDATIONS 

1.  The  Navy  and  Marine  Corps  should  use  the  revised  test  battery  to  aid  in  selecting 
recruiters. 

2.  The  validity  of  the  battery  should  be  monitored  to  ensure  its  continued 
usefulness, 

3.  The  performance  rating  materials  should  be  distributed  to  recruiter  supervisors 
and  field  recruiters  to  aid  in  training  and  self-development. 
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